Missing Data and Imputation

Michael Underwood

Elizabeth Subject-Scott

Javier Estrada

What is Missing Data?

Missing data occurs when there are missing values in a dataset
- Can be intentional or unintentional
Missing data is classified into 3 different categories:
- Missing Completely At Random (MCAR)
- Missing At Random (MAR)
- Missing Not At Random (MNAR)

_{X are the completely observed variables.}

_{Y are the partly missing variables.}

_{Z is the component of the cause of missingness unrelated to X and Y.}

_{R is the missingness.}

Methods to Handle Missing Data

Likelihood Bayesian Method
- Predicts missing values based on a previous predictive distribution.
Weighting Method
- Uses weights from available data to adjust for missing values.
Imputation Method
- Uses estimates from original data to determine missing values

Deleting Missing Data

When type is MCAR and the amount of missing data is small, deletion can be used.
2 Types
- Listwise deletion occurs when the entire observation is removed.
- Pairwise deletion occurs when the variable of an observation is removed.
Deleting missing data can lead to the loss of important information regarding your dataset and is not recommended.

Preferred Method:

Imputation
- 2 Types
  - Single Imputation
    - Only one estimate is used to replace the missing data.
  - Multiple Imputation
    - Various estimates are used to replace the missing data by creating mulitple versions of the original dataset.

Single or Univariate Imputation

Methods include:
- Using the mean to replace a missing value.
  - The problem with this method is that it reduces the variance which leads to a smaller confidence interval.
- Last Observation Carried Forward (LOCF) replaces a missing value with a previously observed value (the most recent value is carried forward).
  - The problem with this method is that it assumes that the previous observed value is perpetual, when in reality that may not be the case.

Multiple Imputation

A set of m plausible values are generated for each unobserved data point, resulting in M complete data sets.
The new values are randomly drawn from predictive distributions either through joint modeling (JM) or fully conditionalspecification (FCS).
It is then analyzed and the results are combined, or pooled together, to obtain a single value for the missing data.
Multiple imputation by chained methods (MICE) is the most common and preferred method of multiple imputation.

_{Rubin’s Rules: Average the estimates across m estimates.}

_{Calculate the standard errors and variance of m estimates.}

_{Combine using an adjustment term (1+1/m).}

Other Methods of Imputation

Regression Imputation is based on a linear regression model. Missing data is randomly drawn from a conditional distribution when variables are continuous and from a logistic regression model when they are categorical.
Predictive Mean Matching is also based on a linear regression model. The approach is the same as regression imputation except instead of random draws from a conditional distribution, missing values are based on predicted values of the outcome variable.

Hot Deck (HD) imputation is when a missing value is replaced by an observed response of a similar unit, also known as the donor. It can be either random or deterministic, which is based on a metric or value. It does not rely on model fitting.
Stochastic Regression (SR) Imputation is an extension of regression imputation. The process is the same but a residual term from the normal distribution of the regression of the predictor outcome is added to the imputed value. This maintains the variability of the data.

Random Forest (RF) Imputation is based on machine learning algorithms. Missing values are first replaced with the mean or mode of that particular variable and then the dataset is split into a training set and a prediction set. The missing values are then replaced with predictions from these sets. This type of imputation can be used on continuous or categorical variables with complex interactions.

Methodology

Point of multiple imputation
- Not in creating artificial values for the missing data
- Intended to be a tool in analyzing data with missing values
- Developed by Donald Rubin, Harvard Professor in the 70s
  - Previous method was not correct
  - Impossible to calculate missing data with certainty
  - It did not take into account the estimates in error in the predictions. He recommended creating multiple datasets, based on the Bayesian method, and focused on the variances of these datasets to help

According to Rubin’s Rule, in multiple imputation missing values are replaced by m imputed values which result in M complete datasets. For each of the M datasets, an estimate of \(\theta\) is acquired. Let \({\hat{\theta}}_{m}\) and \({\hat{\phi}}_{m}\) be an estimator of the variance of \({\hat{\theta}}_{m}\) based on the Mth complete dataset.

The combined estimator of \(\theta\) is given by:

\[{\hat{\theta}}_{M} = \displaystyle \frac{1}{M}\sum_{m = 1}^{M} {\hat{\theta}}_{m}\] That is 1 / number of imputed datasets times the sum of the difference in each imputed dataset

The proposed variance estimator of \({\hat{\theta}}_{M}\) is given by:

\[{\hat{\Phi}}_{M} = {\overline{\phi}}_{M} + (1+\displaystyle \frac{1}{M})B_{M}\]

with a correction factor of \((1+\displaystyle \frac{1}{M})\),

where the average within imputation variance is (normal standard error):

\[{\overline{\phi}}_{M} = \displaystyle \frac{1}{M}\sum_{m = 1}^{M}{\hat{\phi}}_m\]

and the between imputation variance is (how much error because of imputation process):

\[B_{M} = \displaystyle \frac{1}{M-1}\sum_{m = 1}^{M}({\hat{\theta}}_{m}-{\overline{\theta}}_{M})^{2}\]

Assumptions for Multiple Imputation

Observed data follow a normal distribution. If not normal, a transformation is first needed.
Missing data are classified as MAR, which is the probability that a missing value depends only on observed values and not unobserved values.
The parameters \({\theta}\) of the data model and the parameters \({\phi}\) of the model for the missing values are distinct. That is, knowing the values of \({\theta}\) does not provide any information about \({\phi}\).

Steps in Multiple Imputation:

Step 1: Impute missing data
Step 2: Run regression models on all imputation sets
Step 3: Pooling regression results into one regression result

MICE in R

MICE = Multivariate Imputation by Chained Equations. - mice() method in R.

The method creates multiple imputations (replacement values) for multivariate missing data.

The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data. - This can be doing using the imputation regressions of the mice() method such as: - predictive mean matching (numeric data) - logistic regression imputation (binary data with two factorial levels) - polytomous regression imputation for unordered categorical data (factor > 2 levels) - proportional odds model for (ordered, > 2 levels).

Dataset

King County, Seattle Home Sale Prices between 2014 and 2015
- Contains the sale prices of 21,613 houses
- The original dataset contained 21 columns with various selling attributes.
  - For the purpose of this project, we have condensed the variables to the following 4:
    - sales price of home
    - number of bedrooms
    - number of bathrooms
    - square footage of living space.

# laod library
library(mice, warn.conflicts=FALSE)

# load data
original = read.csv("kc_house_data.csv")

#first 5 rows
head(original, 1)

   price bedrooms bathrooms sqft_living
1 221900        3         1        1180

# duplicate dataset
house <- original

# replace some values with NA to create missing data
set.seed(10)
house[sample(1:nrow(house), 200), "sqft_living"] <- NA
house[sample(1:nrow(house), 200), "bedrooms"] <- NA
house[sample(1:nrow(house), 200), "bathrooms"] <- NA
house[sample(1:nrow(house), 100), "price"] <- NA

# check for missing data
sapply(house, function(x) sum(is.na(x)))

      price    bedrooms   bathrooms sqft_living 
        100         200         200         200

Our dataset now has 700 NA/missing values. This equates to about 3% of the data (700/21,613).

Visualizations

# check the missingness patterns
md.pattern(house, rotate.names = TRUE)

      price bedrooms bathrooms sqft_living    
20920     1        1         1           1   0
195       1        1         1           0   1
196       1        1         0           1   1
2         1        1         0           0   2
197       1        0         1           1   1
2         1        0         1           0   2
1         1        0         0           1   2
98        0        1         1           1   1
1         0        1         1           0   2
1         0        1         0           1   2
        100      200       200         200 700